1 

1 

2 
3 
4 
5 
6 
7 
8 
9 
10 
11 
12 
13 
14 
15 
16 
17 
18 
19 
20 
21 
22 
23 
24 
25 




In the Claims 

Claims 1-41 were originally filed. 

Claim 24 is amended. 

Accordingly, claims 1-41 are pending. 



Clean Version Of The Pending Claims Under 37 C.F.R. S 1.121(c)(3): 

Claims 1-41 now pending, are submitted below in accordance with 37 
C.F.R. §1.12 1(c)(3), which presents a clean version of the entire set of pending 
claims. 

1. A method combrising: 

initiating a search fpr images based on at least one query keyword in a 
query; and 

identifying, during fhe search, first images having associated keywords that 
match the query keywom and second images that contain low-level features 
similar to those of the first images. 

2. A methodps recited in claim 1, further comprising ranking the first 
and second images. 



A method as recited in claim 1, further comprising presenting tlie 



first and second images. 



/ 



4, A met|hod as recited in claim 1 , further comprising: 
presenting ttte first and second images to a user; and 
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monitoring feedback from the user as to which of the first and S9rond 
images are relevant to the query. 
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5. A method as recited in claim 1, further comprising: 
presenting the first and second images to a user; 

receiving feedback from the user as to whether the firs^ and second images 
are relevant to the query; and 

learning how the first and second images are /dentified based on the 
feedback from the user. 

6. A method as recited in claim 1, further comprising: 
presenting the first and second images t</a user; 

receiving feedback from the user as p which of the first and second images 
are relevant to the query; and 

refining the search to identifV additional images that contain low-level 
features similar to those of the imag^ indicated by the user as being relevant to the 
query. 
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7. A method as recJited in claim 1, further comprising: 
presenting the first and second images to a user; 

receiving feedbaok from the user as to which of the first and second images 
are relevant to the quety; and 

assigning a large weight to an association between the query keyword and 
the images deemed relevant by the user. 
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8. A method as recited in claim 7, further compr^ing grouping the low- 
level features of the images deemed relevant by the user. 

9. A method as recited in claim 1, further comprising: 
presenting the first and second images to a usp; 

receiving feedback from the user identifying an example image as less 
relevant or irrelevant to the query for refinementyof the search; and 

assigning a small weight to an associapon between the query keyword and 
the example image. 

10. A method as recited in/claim 9, further comprising identifying 
additional images with low-level features similar to those of the example image. 

11. A computer readable medium having computer-executable 
instructions that, when executea on a processor, perform the method as recited in 
claim 1 . 



12. A method comprising: 
)/b 



permitting entry af both keyword-based queries and content-based queries; 
finding images Aising both semantic-based image retrieval and low-level 
feature-based image retrieval; 

presenting the images to a user so that the user can indicate whether the 
images are relevant; and 

. / 

conducting semantic-based relevance feedback and low-level feature-based 
relevance feedback in an integrated fashion. 
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13. A method as recited in claim 12, furtl^r comprising ranking the 
images. 

14. A method as recited in claim 12, mrther comprising using images 
indicated as being relevant to find additional images. 
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15. A computer readable meaium having computer-executable 
instructions that, when executed on a processor, perform the method as recited in 



claim 12. 
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16. A method comprising: . 

associating keywords with ini^ges to form keyword-image links; 
assigning weights to the keyword-image links; 

presenting a result set of/images obtained from an image retrieval search 
based on a query; 

receiving feedback^nbAi a [user as to whether the images in the result set are 
relevant to the query; and 

modifying the weiglits according to the user feedback. 

17. A method as recited in claim 16, wherein the modifying comprises 
increasing the weight of a keyword-image link for images deemed by the user as 
more relevant to the query. 



25 
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18. A method as recited in clajm 16, wherein the modifying comprises 
decreasing the weight of a k^^or^Tiniage Hnk for images deemed by the user as 
less relevant to the query^ 

19. A /computer readable medium having computer-executable 
instructioijsahat, when executed on a processor, perform the method as recited in 
clairD46. 

20. A method comprising: 
presenting a result seft of images that are retumed from an image retrieval 

search of a query having at least one keyword; 

monitoring feedbac^ from a user as to whether the images in the result set 
are relevant to the query; 

in an event that the user selects at least one image as being relevant to the 
query, associating the keyword in the query with the selected image to form a first 
keyword-image association and assigning a comparatively large weight to the first 
keyword-image association; and 

in an event th^ the user identifies an example image for refinement of the 
search, associating tHe keyword in the query with the example image to form a 
second keyword-image association and assigning a comparatively small weight to 
the second keyword-Image association. 



21. A method as recited in claim 20, further comprising conducting both 
content-based image retrieval and semantic-based image retrieval. 
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22. A method as recited in claim 20, further comprising presenting the 
result set of images in a user interface, the user interface^ facilitating the user 
feedback by allowing the user to indicate which images/are more relevant and 
which images are less relevant. 

23. A computer readable medium having computer-executable 
instructions that, when executed on a processor, perform the method as recited in 
claim 20. 

24. (Once Amended) A method comprising: 

computing, for each category, a repj^esentative feature vectors of a set of 
existing images within the category; 

determining a set of representatyC^e keywords that are associated with the 
existing images in each category; 

comparing, for each new imsige, the low-level feature vectors of the new 
image to the representative feature^ vectors of the existing images in each category 
to identify a closest matching category; and 

labeling the new image/with the set of representative keywords associated 
with the closest matching category. 

25. A method yas recited in claim 24, further comprising using use 
feedback to selectively add and/or remove keywords from the new image. 

26. A method as recited in claim 24, further comprising: 
placing the labeled new images into a holding category; 
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evaluating the labeled new images in the holding categow to determine if 
any of the keywords associated with the labeled new /image match the 
representative keywords from each category; and 

assigning the labeled new image to the categor/ that best matches the 
keywords associated with the labeled new image. 
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27. An image retrieval system comprisim 
a query handler to handle both keyword-based queries having one or more 

search keywords and content-based queries having one or more low-level features 
of an image; and 

a feature and semantic mat^her^^^C^njkify at least one of (1) first images 
having keywords that match the sekrch l^eyxySrds from a keyword-based query, and 
(2) second images having low-level f/atures similar to the low-level features of a 
content-based query. 

28. An image retrieval^fystem as recited in claim 27, wherein the feature 
and semantic matcher ranks the images. 

29. An image retrieval system as recited in claim 27, wherein the query 
handler comprises a natiaral language parser. 

30. An injage retrieval system as recited in claim 27, wherein the query 
handler comprises^ 

a parser to parse text-based queries; and 
a concept hierarchy to define various categories of images. 
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31. An image retrieval system as recited in claim 27yfurther comprising 
a user interface to present the images identified by the ^ature and semantic 
matcher. 
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32. An image retrieval system as recited in claim 27, further comprising: 
a user interface to present the images identifieaby the feature and semantic 

matcher to a user, the user interface allowing ther user to indicate whether the 
images are relevant to the query; and 

a feedback analyzer to train the image retrieval system based on user 
feedback as to relevancy. 

33. An image retrieval system ars recited in claim 27, further comprising: 
a user interface to present the iniages identified by the feature and semantic 

matcher to a user, the user interface allowing the user to identify an example 
image; and 

the feature and semantic/matcher being configured to identify additional 
images that contain low-level features similar to those of the example image. 

34. An image retrieval system as recited in claim 27, further comprising: 
a user interface id present the images identified by the feature and semantic 

matcher to a user, the user interface allowing the user to identify which images are 
relevant to a particular search query; and 

a feedback analyzer to assign a large weight to an association between the 
search keywords and the images identified as relevant by the user. 
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35. An image retrieval system as recitefl in claim 34, wherein the 
feedback analyzer groups the low-level feature^ of the images identified as 
relevant by the user. 

36. An image retrieval system as recited in claim 21, further comprising: 
a user interface to present the images /dentified by the feature and semantic 

matcher to a user, the user interface allowiitg the user to identify an example image 
as being less relevant or irrelevant to the query; and 

a feedback analyzer to assign a small weight to an association between the 
search keywords and the example image. 

37. An image retrieval sy/em as recited in claim 36, wherein the feature 
and semantic matcher identifies ^ditional images with low-level features similar 
to those of the example image. 



38. A database structure stored on one or more computer-readable media 
comprising: 

multiple image files/ 
multiple keywords/ and 

a semantic network to associate the keywords with the image files, the 
semantic network detfining individual keyword-image links that associate a 
particular keyword with a particular image file, each keyword-image link having a 
weight indicative of how relevant the particular keyword is to the particular image 
file. 
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39. A computer-readable medium havmg computer-executable 
instructions that, when executed, direct a computer to:/ 

find images using both semantic-based image retrieval and low-level 
feature-based image retrieval; / 

present the images to a user so that the>user can indicate whether the images 
are relevant; and / 

concurrently conduct semantic-based relevance feedback and low-level 
feature-based relevance feedback. / 

40. A program as recited in claim 39, further comprising computer- 
executable instructions that, ^en executed, direct a computer to rank the images. 

41. An information retrieval program, embodied on the computer- 
readable medium, comprising the computer-executable instructions of claim 39. 
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